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! . St  a turner, i 

A system  must  operate  for  I units  of  time,  A certain  component 
is  essential  for  the  system  to  he  operative.  When  it  tails  it  must  be 
rlaced.  Hoi  there  ai  , ; - a be  u ed; 

some  types  are  more  costly  but  tend  to  last  longer  --  others  are  cheaper 
but  are  not  likely  to  last  as  long. 

The  general  problem  is  to  assign  the  initial  component  type  and 
all  subsequent  replacement;  so  as  . o minimize  the  expected  cost  ol:  pro- 
viding an  operative  component  for  the  t the  system. 

For  example,  a car  owner  who  keeps  his  car  for  a random  length  of 
time  must  from  time  i'<  t.  ■ tn«.  i •-•place  the  battery.  A number  o;  possible 
replacement  types  exist.  Longer  las;  ..ng  batteries  cos!  more.  However, 
tie  trade-in  price  for  the  car  will  not  reflect  the  t 'ti  ;■  condition  oi 
battery  in  use.  Thus,  the  problem  of  type  of  battery  to  use  as  a replace 
meat  naturally  pi -.sent s itself  Lf  the.  car  owner  is  concerned  with  operating 
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costa. 


It  will  he  soon  that  making  optimal  rcpla  on, out  decisions  may 


produce  only  modest  savings  for  the  individual  ur  owner  ; or  system 
operator).  However  tor  the  large  fleet  owner  such  as  the  Federal  or 
State  government,  the  cumulative  advantage  of  making  optimal  decisions 
may  be  very  large. 

intuition  would  suggest  that  when  the  system  is  relatively  new 
and  a replacement  is  necessary  it  would  be  rational  to  use  an  expensive 
type  providing  the  additional  component  Life  bought  is  worth  the  extra 
price;  similarly  when  the  system  is  old,  an  inexpensive  type  should  be 
used.  Consideration  of  this  probLem  is  intended  to  make  this  intuition 
more  precise  and  to  explore  under  what  conditions  it  is,  in  fact  valid. 

The  problem  can  be  viewed  as  one  in  discrete-  Lime  parameter 
negative  dynamic  programming.  The  state  of  the  system  is  its  age  when  a 
replacement  is  necessary.  The  pussibLe  decisions  are  the  different  types 
of  components  to  use.  The  costs  are  the  costs  of  the  types  used.  Laws 
of  motion  are  determined  by  the  life  distribution  functions  of  the  different 
component  types.  The  life  of  the  system  is  T. 

The  paper  by  Strauch  [ ji  ! covers  foundations.  An  optimal  policy 
exists:  it  will  be  non- randomized  and  stationary;  the  optimal  value  func- 
tion will  satisfy  the  optimality  equation  and  any  value  function  corre- 
sponding to  a policy  that  satisfies  the  optimality  equation  will  be  an 
optimal  value  function. 


Fixed  Horizon  Ca  Sc 


o 

We  assume,  here  that  it  is  known  precisely  when  the  entire  system 
will  be  terminated.  Instead  ol  using  the  ag  ol  the  sysLeir,  it  is  con- 
venient to  let  t the  time,  until  system  termination  be  th*  state  ol"  the 
system  when  a replacement  is  necessary. 

We  assume  throughout  that  the  distribution  of  life  of  a type  i 
component  is  exponential  with  mean  A.,1'.  In  most  cases  thi  s is  not  a 
realistic  assumption.  However,,  as  in  the  usual  justification,  we  consider 
the  assumption  to  be  a first  approximation. 

Let  C denote  the  cost  of  a type  i replacement  and  V( t) 
denote  the  expected  cost  of  ail  remaining  replacements  given  t units  of 
time  until  termination  and  an  optimal  replacement  policy  is  used.  Assume 
that  the  types  are  indexed  so  thaL  A.C,  > A,-C„  > ■ ■•  > ?v  C . 

The  main  result  that  appears  in  [1]  is  as  follows:  V( t)  is 

piece-wise  linear  and  concave.  An  optimal  policy  car.  be  characterized 
as  n intervals  (some  of  which  may  be  vacuous)  where  a type  i component 
is  used  for  replacement  when  t units  remain  until  termination  and  t 
is  in  the  i interval  from  the  origin. 

The  proof  of  the  theorem  exploits  the  optimality  equation 

1 f ~^iX  I 

1)  Vt  t)  min  C.  + j V(  t-x)  A.  e 1 dx  t 0 

i ( 1 0 1 i 

Once  it  is  established  that  V(  t)  is  continuous  at  all  t > 0,  the 
expression  ; ( t) , say.  within  the  brackets  of  (l)  is  easily  shown  to 
be  differentiable  for  each  i.  Thus,  if  i is  optimal  at  t,  on 
differentiation  ii  is  seen  that 

3 


q(t) 


> c.  . 

1 i 


If  i is  uniquely  optimal  at  t then  from  (1)  ami  continuity  it  is 
optimal  in  the  interval  ( t,  t+e)  for  some  e a-  0.  If  i and  j are 
both  optimal  at  t then  from  (1.)  and  the  above  derivative  it  follows 
that  within  (t,  t+e),  for  some  t > 0f  the  type  with  the  smallest  AC 
is  optimal.  The  stated  structure  of  the  optimal  policy  follows. 

An  algorithm  for  obtaining  the  values  of  t where  the  optimal 
replacement  type  changes  is  a corollary  of  the  optimal  policy  structure. 

The  values  are  obtained  recursively.  Let  it,  denote  the  optimal  policy 
when  only  types  I,  ■ k are  available;  is  obvious.  Let  V,( t)  be  the 

value  function  using  jt^.  Since  the  optimal  value  function  is  continuous 

* 

the  value  t satisfying 


(-J) 


V*  > Ck+1 


vk(t  'x>  w exp(-Ak+lx)  dx 


is  that  value  of  time  until  termination  such  that  r,  , uses  type  k+1 

kfl 


* *• 

for  all  t > t ; lor  t < t f 


coincides  with  it,  . if  n 2. 
k-t  t k ’ 


C ' iCl  > 'A- 


Y''  i * cr^T,l 

:1?1  ' C2^j 


, rC.>,  - C0?„ 

loK 


The  algorithm  automatica L Ly  excludes  those  types  which  would  never 

be  used;  their  corresponding  intervals  would  be  vacuous.  If  > C^} 

A,  > a.  (or  C,  > C..  A,  A.),  the  case  where  a larger  cost  purchases 

J i J l * j — i'  ' 


a shorter  expected  life  then  type  j is  never  used.  This  is  shown 
formally  in  [lj  by  comparing  the  policy  n.  of  first  using  type  j and 
replacing  optimally  thereafter  with  the  policy  i<  . of  first  using  type  i 

and  replacing  optimally  thereafter.  If  V and  V are  the  respective 

jt . n . 

.1  i- 

value  functions  then  for  ail  t > 0. 

(3)  V ( t)  - V ( t)  = (C.-C.)  + / V(  t-x)  ( A.  exp[-A.x]  - A.  exp[-A.xj)  dx. 
Jtj  J r 0 ' \ J J i J-  / 

The  first  term  on  the  right  of  (3)  is  non-negative  (positive).  Because  the 
exponential  distribution  with  the  greater  mean  dominates,  and  since 
V(t-x)  is  decreasing  in  x the  second  term  is  also  positive  (non-negative). 
Thus  V - V >0  and  it  is  never  optimal  to  replace  with  type  j. 

JT  . JT  . 

J 1 

The  case  of  n = 2 suggests  the  conjecture  made  in  [ l]  that  a 

sufficient  condition  for  type  j never  to  be  used  is  that  C > C.  and 

J i 

C.A.  > C.A.  (or  C.  > C.  and  C.A.  > C.A.).  The  comparison  (3)  fails 
J J - t i J - t jj  xi'  r ' ' 

to  be  conclusive  because  the  condition  A.  > is  not  implied  and  a 

.1  i 

useful  estimate  of  the  integral  in  (3)  is  not  available.  Smith  [2]  uses 

an  ingenious  comparison  to  avoid  the  necessary  evaluation  of  an  integral 

involving  the  function  V.  Considering  the  case  where  C.A.  > C.A.  but 

J J t t 

A.  < A.  (hence.  C.  > C.).  Smith  compares  jr . with  • . a specified 

j r ' ’ j i/  r j 

randomized  policy,  n initially  uses  type  i.  At  each  replacement  a 
chance  experiment  is  carried  out  so  that  type  i is  used  again  with 
probability  (A^-A.)/A..  When  for  the  first  time  Lhe  chance  experiment 
indicates  that  type  i is  not  to  be  used  then  that  replacement  and  all 
subsequent  ones  are  made  optimally.  The  usefulness  of  this  device  appears 


in  the  fact  Chat  t lie  length  of  time  until  a replacement  other  than  type  i 
is  made  lias  an  exponential  distribution  with  mean  a.*’,  the  same  as 
would  be  the  distribution  of  the  length  oi  time  unLil  the  second  component 
is  introduced  under  policy  n . . Thus,  in  the  comparison  between  V 

J (j 

and  V~  analogous  to  ( Lite  integral  vanishes  and 
i 


V ( t)  - V~  ( t)  (C.-C.)  - M.  , 

Jt  . Tt  . I L J 1 

J 1 


where  M.  is  the  expected  cost  of  using  type  i components  as  a result 

of  the  chance  experiments.  M.  can  be  calculated,  and  It  then  follows 

1 


V ( t)  - V~  ( t)  > 0 

it . Jt . ' — ’ 

J i 


for  all  t > 0 . 


Given  the  foregoing  it  is  then  possible  to  assert,  without  loss  of 
generality, that  the  types  are  indexed  so  that  < C.  • • • < and 
Ci*i  > Cg Ap  > •••  > C^An,  for  otherwise  some  types  could  be  excluded  in 
advance.  Given  this  ordering, it  is  easily  argued  that  if  t is  small 
enough, type  1 should  be  used  as  a replacement.  Thus  if  no  value  of  t 
satisfies  (2),  one  can  conclude  inductively  that  type  k< 1 is  never  used 
and  its  interval  of  optimality  is  vacuous. 


The  optimal  policy  structure  seems  to  reinforce  intuition.  However 
it  is  not  difficult  to  produce  counter-examples,  deviating  from  the  assump- 
tion of  exponential  component  life  where  such  a structure  is  not  optimal. 
In  fact,  the  exponential  distributions  may  belong  to  a minority  of 
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distribm  ions  lur  which  this  structure  holds.  Whether  t U exponent  ial 


assumption  nevertheless,  rt  sents  ■ * (nation  tin 

to  be  investigated. 


Rantloi.i  dor  xxon  Cast 

One  serious  limitation  rm  the  model  cons idured  to  L h::;  point  is 
the  assumption  that  the  entire  system  (ol  which  'ur  component  of  interest 
is  a part)  will  be  discarded  at  a fixed  time.  This  assumption  may  hold 
when  a strict  disposal  policy  is  in  effect  which  specif  res  at  what  time 
the  system  will  be  abandoned.  However,  even  when  such  a policy  is  in 
eftect  there  is  usually  a provision  lor  disposing  of  Lhe  system  if  it 
Is  prior  to  the  mandatory  time  of  disposaf.  Within  this  context  the 
d horizon  model  is  extended  in  [ 1 ] . 

It  is  assumed  that  the  system  will  be  disposed  at  a fixed  time  or 
before  if  a failure  occurs.  The  length  of  time  until  failure  is  exponent ia 
distributed  with  mean  Ci  ^ (say).  Thus  the  length  of  time  until  disposal 
is  a truncated  exponential  distribution.  Within  this  model  it  is  still 
convenient  to  let  the  state  of  the  system  be  designated  as  the  maximum 
amounL  of  time  remaining  until  Lhe  system  terminates.  Analogous  to  ( 1) , 
interpreting  V( t ) as  before,  we  have  Lhe  optimality  equation 

1 ,fc  -ax  ■ A j x j 

(4)  V(  t.)  = min  ;C  +■  I e V(  t-x)  A.  e dx  J L > 0 . 

i I 1 0 ' I 
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Employing  the  same  chain  of  reasoning  as  in  the  fixed  horizon 
case;  the  expression,  |>  (t)  , within  the  brackets  of  (4)  has  a 
derivative  which  Ls  given  by 

+'!(t)  - (atAi)C.  - QV(t)  , 

if  i is  optimaL  at  t.  The  interval  structure  of  an  optimal  policy 
still  holds.  V is  still  a concave  function.  However,  the  piecewise 
linearity  gives  way  to  the  appropriate  solution  of  the  differential 
equat ion 

V'(t)  = (Qh-Ai)C.  - OV(t) 

over  the  interval  for  which  type  i is  the  optimal  replacement. 

It  should  be  remarked  that  this  model  is  mathematically  identical 
to  the  fixed  horizon  case  with  discounted  cost  criterion.  Hence,  the 
interval  structure  for  an  optimal  policy  holds  when  costs  are  discounted. 

It  is  to  be  expected  that  the  assumptions  regarding  the  distribution 
of  T the  system  iife,  can  be  further  weakened.  To  this  end  suppose  F 
is  the  distribution  function  of  T.  It  is  necessary,  now,  to  define  the 
state  of  the  system  as,  s,  the  length  of  time  the  system  has  oeen  operating, 
rather  than  the  time  remaining  until  termination.  Let  Uf s)  denote  the 
expected  future  cost  of  supplying  the  needed  component  given  the  system 
has  been  operating  for  s units  of  time,  a replacement  is  necessary. 
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r 


and  all  replacements  will  be  made-  optimally.  The  opLiroality  equation 
that  U(  s)  satisfies  is 


(5) 


Ll(  s)  min.C.  i 


rAix 


L'( 


i - 


i.( a 1 x 

s) 


a-  dx 


nuntep^  s)  J , 


where  c,  . ( i 1,  n)  is  the  expression  within  the  brackets  of  (9). 

Some  limitations  on  F must  be  mposed  if  the  i irr.e  interval 
structure  for  an  optimal  policy  as  obtained  for  the  fixed  horizon  case 
is  to  prevail.  One  can  easily  construct  examples  of  F where  the  optimal 
policy  will  not  have  the  interval  structure.  However,  if  F has  an 
increasing  failure  rate  (IFR)  the  property  should  held.  Below  is  a sketch 
of  the  proof  that  this  is,  indue d(  true  for  the  case  where  n = 2 and  F 
is  strictly  IFR.  in  addition  to  F being  IFR  it  is  also  assumed  that  F 
is  continuous  except,  perhaps,  at  x where  F(>Iq)  = 1. 

As  before,  it  can,  without  loss  of  generality,  be  assumed  that 
> O.  /.j,  'A  C,,.  Using  the  assumption  of  continuity  it  is  straight- 
forward to  prove  that  U(  s)  is  continuous  at  all  s < x(1.  Then  q_(s), 
i _•  1 'd  is  differentiable  and  if  t is  optimal  at  s 


o !(s)  r(  s)  U(s)  - (A.  + r(s))C.  , 
i i ' t 


where  r(  s)  is  the  failure  rate  function  of  F.  Since  r is  increasing 
there  is  a value  s such  that  either 
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By  cont inuity , if  s * is  an  age  where  the  optimal  replacement  type 
switches, then  boLli  types  are  optimal  at  s*.  Howev  -r,  by  (6)  in  the 
interval  s < s only  a switch  from  type  2 to  type  1 is  possible  and  in 
the  interval  s > s only  a switch  from  type  1 to  type  2 is  possible.  In 
the  latter  case  type  2 would  be  optimal  for  all  system  ages  s > s*  where 
s*  > s and  s*  is  the  age  at  which  the  switch  from  type  1 to  type  2 
takes  place.  However,  a direct  comparison  with  the  policy  of  always 
using  type  1 for  all  s > s*  shows  that  always  replacing  with  type  2 
for  s > s*  cannot  be  optimal.  Hence,  a switch  from  type  1 to  type  2 
for  any  s > s cannot  be  optimal.  The  remaining  possibilities  are 
consistent  with  what  was  to  be  proved. 

Recently,  Derman  and  Smith  have  extended  this  result  to  n > 2. 

An  algorithm  is  also  suggested  for  obtaining  switching  points. 
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Bounds 


An  upper  bound  on  U(  s)  is  easily  obtained  by  evaluating  the 
policy  that.  always  uses  type  n as  a replacement.  Thus 

U(  s)  < A C E(t|t  > s)  t C 

To  obtain  a lower  bound  a modified  problem  is  considered  in  [ l 1 . The 

modification  consists  in  allowing  a rebate  of  the  amount  C.-C|  ii  d 

type  j component  is  in  use  at  the  time  the  system  terminates.  The 

expected  optimal  cost  for  this  problem  is  no  larger  than  the  expecLcd 

optimal  cost  for  the  original  problem.  However,  for  the  modified  problem 

it  is  optimal  lo  use  a type  n component  for  every  replacement  (assuming 

AC  < •••  < A.C.).  This,  a lower  bound  for  U(  s)  is 
n n 11  ’ 

U(s)  > A C E(t!t  > s)  i-  C.  . 

If  Cn-C^  is  small  the  question  of  optimal  replacement  has  little 
practical  interest  unless  many  systems  are  in  operation  so  that  the 
optimality  pay-off  is  manifested  in  the  aggregate. 

Problem  Variations 

Besides  the  modified  problem  alluded  to  in  the  previous  section 
several  other  modifications  are  considered  in  [ 1 J for  the  tixed  horizon 


1L 


case . 


One  variation  assumes  that  there  are  two  replacenv  " t types.  An 
infinite  number  of  type  2 components  are  available;  however  only  one  type  1 
component  is  available.  It  is  shown  that  the  same  two  interval  structure 
is  optimal.  The  switching  point  is  evaluated  but  is  not  the  same  as 
the  original  n - 2 problem.  Optimal  stopping  theory  is  the  method 
used  to  establish  the  result.  The  method,  although  not  the  switching 
point,  is  free  of  the  exponential  component  life  assumption  allowing  one 
to  obtain  the  interval  structure  for  an  optima:  policy  under  weaker 
assumptions. 

The  reverse  situation  where  an  infinite  number  of  type  1 components 
are  available  and  only  one  of  type  2 also  has  the  same  optimal  policy 
structure.  The  switching  po.inL,  as  would  be  expected,  is  the  same  as  in 
the  original  problem.  The  method,  here,  is  to  guess  that  the  optimal 
policy  is  of  this  form  and  verify  that  value  function  satisfies  the 
optimality  equation. 

The  problem  with  the  rebate  that  yields  the  lower  bound  discussed 
in  the  previous  section  assumes,  except  for  at  least  one  type,  that 
there  are  only  a finite  number  of  replacements.  That  is  only  one  type 
must  have  an  infinite  supply  in  order  that  a replacement  is  always 
available.  The  horizon  can  be  of  random  length  as  well  as  fixed.  The 
optimal  policy  is  to  always  use  the  type  available  that  has  the  smallest 
AC. 
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ON  RENEWAL  DECISIONS 

by 

C.  Herman,  G.  J.  Lieberman  and  S.  Ross 


1 A system  must  operate  for  T units  of  time.  A certain  component  is 

essential  for  the  system  to  be  operative.  When  it  fails  it  must  be  replaced. 

I 

! However,  there  are  n types  of  components  that  can  be  used;  some  types  are 
I more  costly  but  tend  to  last  longer  — others  are  cheaper  but  are  not  likely 
i to  last  as  long. 

The  general  problem  is  to  assign  the  initial  component  type  and  all 
subsequent  replacements  so  as  to  minimize  the  expected  cost  of  providing 
an  operative  component  for  the  life  of  the  system. 

Intuition  would  suggest  that  when  the  system  is  relatively  new  and 
a replacement  is  necessary  it  would  be  rational  to  use  an  expensive  type 
providing  the  additional  component  life  bought  is  worth  the  extra  price; 
similarly  when  the  system  is  old,  an  inexpensive  type  should  be  used. 
Consideration  of  this  problem  is  intended  to  make  this  intuition  more  precise 
and  to  explore  under  what  conditions  it  is,  in  fact,  valid. 
r-,:  The  problem  can  be  viewed  as  one  in  discrete  lime  parameter  negative 

dynamic  programming.  ^ The  state  of  the  system  is  its  age  when  a replacement 
is  necessary.  The  possible  decisions  are  the  different  types  of  components 
to  use.  The  costs  are  the  costs  of  the  types  used.  Laws  of  motion  are 
determined  by  the  life  distribution  functions  of  the  different  component 
types.  The  life  of  the  system  is  T. 

This  problem  was  originally  considered  by  the  authors  in  "A  Renewal 
Decision  Problem".  This  paper  summarizes  these  results  and  considers 
various  extensions. 
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